Managed ml diagnostics and xpk integration #801

DannyLiCom · 2025-11-04T03:27:19Z

Description

Adding Webhooks, Injector, and connection-operator installations during cluster creation.

Testing

Test with the cluster create command and add the flag --managed-mldiagnostics. The cluster will then install the required ML Diagnostics components: cert-manager, the injection-webhook, and the connection-operator.
like this: https://paste.googleplex.com/6009656740806656

DannyLiCom · 2025-11-04T03:31:08Z

@xibinliu and @Shuang-cnt please review it.

jamOne- · 2025-11-04T08:34:26Z

What's the justification for this change? Why are we adding this for every cluster?
What's diagon?
Could you cover the changes with unit tests please? https://github.com/AI-Hypercomputer/xpk/blob/main/docs/testing.md#unit-test

jamOne- · 2025-11-04T08:35:56Z

Also, please merge the current main branch, because the PR changes are mixed with the sub-slicing changes, making it harder to review.

DannyLiCom · 2025-11-04T08:59:47Z

Also, please merge the current main branch, because the PR changes are mixed with the sub-slicing changes, making it harder to review.

When I run git checkout -b, should I branch off of main or develop? My file changes only touch 3 files, but if I merge into main, there are now 6 changes.

DannyLiCom · 2025-11-04T09:04:13Z

What's the justification for this change? Why are we adding this for every cluster?

What's diagon?

Could you cover the changes with unit tests please? https://github.com/AI-Hypercomputer/xpk/blob/main/docs/testing.md#unit-test

Perhaps @xibinliu needs to explain for it to be clearer.

scaliby · 2025-11-04T09:09:56Z

Also, please merge the current main branch, because the PR changes are mixed with the sub-slicing changes, making it harder to review.

When I run git checkout -b, should I branch off of main or develop? My file changes only touch 3 files, but if I merge into main, there are now 6 changes.

The main. Please just pull latest main and merge it into your branch - that should do the job.

src/xpk/commands/cluster.py

DannyLiCom · 2025-11-05T01:02:38Z

Also, please merge the current main branch, because the PR changes are mixed with the sub-slicing changes, making it harder to review.

When I run git checkout -b, should I branch off of main or develop? My file changes only touch 3 files, but if I merge into main, there are now 6 changes.

The main. Please just pull latest main and merge it into your branch - that should do the job.

Done

DannyLiCom · 2025-11-05T07:30:38Z

What's the justification for this change? Why are we adding this for every cluster?

What's diagon?

Could you cover the changes with unit tests please? https://github.com/AI-Hypercomputer/xpk/blob/main/docs/testing.md#unit-test

ML Diagnostics components

For point 1: Add the --managed-mldiagnostics flag to allow the user to decide whether or not to install these ML Diagnostics components.

DannyLiCom · 2025-11-06T06:14:34Z

@jamOne- @scaliby The unit tests are failing because our installation involves the Kubernetes API server. Does the installation process require unit tests? If so, it will get stuck at get-credentials.

[XPK] ********************************************************************************
[XPK] b'E1106 04:14:54.063296    1983 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"http://localhost:8080/api?timeout=32s\\": dial tcp [::1]:8080: connect: connection refused"\nE1106 04:14:54.063919    1983 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"http://localhost:8080/api?timeout=32s\\": dial tcp [::1]:8080: connect: connection refused"\nE1106 04:14:54.065489    1983 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"http://localhost:8080/api?timeout=32s\\": dial tcp [::1]:8080: connect: connection refused"\nE1106 04:14:54.065944    1983 memcache.go:265] "Unhandled Error" err="couldn\'t get current server API group list: Get \\"http://localhost:8080/api?timeout=32s\\": dial tcp [::1]:8080: connect: connection refused"\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?\n'
[XPK] ********************************************************************************

scaliby · 2025-11-06T10:16:42Z

@DannyLiCom please just mock this behavior on command level, so there will be no actual command execution happening.

src/xpk/commands/cluster.py

src/xpk/commands/managed_ml_diagnostics.py

src/xpk/commands/managed_ml_diagnostics_test.py

src/xpk/parser/cluster.py

src/xpk/commands/managed_ml_diagnostics.py

src/xpk/commands/managed_ml_diagnostics_test.py

…ad.py Add wait_for_deployment_ready() Added unit test update goldens.yaml update goldens.yaml update goldens.yaml Fixed parser/cluster.py update goldens.yaml fixed linter fixed linter pyink Test unit test

DannyLiCom · 2025-11-21T05:36:17Z

Hi @jamOne- @scaliby I think I'm done with the modifications. Please review it again.

src/xpk/commands/managed_ml_diagnostics_test.py

src/xpk/parser/cluster.py

src/xpk/parser/cluster_test.py

DannyLiCom · 2025-11-21T15:13:55Z

@scaliby I don't have the permission to merge.

DannyLiCom requested review from FIoannides, SikaGrr, jamOne- and scaliby as code owners November 4, 2025 03:27

DannyLiCom marked this pull request as draft November 4, 2025 03:27

DannyLiCom added the release-features features label Nov 4, 2025

DannyLiCom changed the base branch from main to develop November 4, 2025 04:02

scaliby changed the base branch from develop to main November 4, 2025 08:13

DannyLiCom force-pushed the diagon-setup-and-workload branch from 681f902 to 20b94c7 Compare November 4, 2025 09:13

scaliby changed the base branch from main to reusable-nightly November 4, 2025 09:31

scaliby changed the base branch from reusable-nightly to main November 4, 2025 09:31

Shuang-cnt reviewed Nov 4, 2025

View reviewed changes

src/xpk/commands/cluster.py Outdated Show resolved Hide resolved

DannyLiCom force-pushed the diagon-setup-and-workload branch 2 times, most recently from 3c46565 to b4aaa13 Compare November 5, 2025 01:01

DannyLiCom changed the title ~~Diagon and XPK Integration~~ Managed ml diagnostics and xpk integration Nov 5, 2025

DannyLiCom force-pushed the diagon-setup-and-workload branch 2 times, most recently from 8997011 to de0c17b Compare November 6, 2025 03:52

DannyLiCom force-pushed the diagon-setup-and-workload branch 3 times, most recently from a41448d to a7c8f82 Compare November 7, 2025 01:40

DannyLiCom force-pushed the diagon-setup-and-workload branch 2 times, most recently from e5ce7be to d761a21 Compare November 19, 2025 18:54

jamOne- requested changes Nov 20, 2025

View reviewed changes

jamOne- reviewed Nov 20, 2025

View reviewed changes

src/xpk/commands/managed_ml_diagnostics_test.py Outdated Show resolved Hide resolved

jamOne- reviewed Nov 20, 2025

View reviewed changes

src/xpk/commands/managed_ml_diagnostics_test.py Show resolved Hide resolved

DannyLiCom added 2 commits November 21, 2025 01:35

Add Diagon installation during cluster creation and modify the worklo…

8d74b69

…ad.py Add wait_for_deployment_ready() Added unit test update goldens.yaml update goldens.yaml update goldens.yaml Fixed parser/cluster.py update goldens.yaml fixed linter fixed linter pyink Test unit test

Resolve conflicts

0a6c9e2

DannyLiCom force-pushed the diagon-setup-and-workload branch 7 times, most recently from bc46f82 to 416e23e Compare November 21, 2025 03:49

Resolve conflicts

4ca23e8

DannyLiCom force-pushed the diagon-setup-and-workload branch from 416e23e to 4ca23e8 Compare November 21, 2025 03:50

DannyLiCom added 3 commits November 21, 2025 04:16

test unit test

e017568

test unit test

1e68060

test unit test

79ccf19

DannyLiCom force-pushed the diagon-setup-and-workload branch from 0cab426 to d27273b Compare November 21, 2025 05:13

jamOne- requested changes Nov 21, 2025

View reviewed changes

src/xpk/commands/managed_ml_diagnostics_test.py Outdated Show resolved Hide resolved

src/xpk/parser/cluster.py Outdated Show resolved Hide resolved

src/xpk/parser/cluster_test.py Outdated Show resolved Hide resolved

deleted set_result_for_command

df4c417

DannyLiCom force-pushed the diagon-setup-and-workload branch from d27273b to df4c417 Compare November 21, 2025 10:44

jamOne- approved these changes Nov 21, 2025

View reviewed changes

jamOne- requested a review from scaliby November 21, 2025 10:50

Merge branch 'main' into diagon-setup-and-workload

52d4693

scaliby approved these changes Nov 21, 2025

View reviewed changes

scaliby merged commit 84f8bad into AI-Hypercomputer:main Nov 21, 2025
14 checks passed

Managed ml diagnostics and xpk integration #801

Managed ml diagnostics and xpk integration #801

Uh oh!

Conversation

DannyLiCom commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

DannyLiCom commented Nov 4, 2025

Uh oh!

jamOne- commented Nov 4, 2025

Uh oh!

jamOne- commented Nov 4, 2025

Uh oh!

DannyLiCom commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DannyLiCom commented Nov 4, 2025

Uh oh!

scaliby commented Nov 4, 2025

Uh oh!

Uh oh!

DannyLiCom commented Nov 5, 2025

Uh oh!

DannyLiCom commented Nov 5, 2025

Uh oh!

DannyLiCom commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scaliby commented Nov 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DannyLiCom commented Nov 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DannyLiCom commented Nov 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DannyLiCom commented Nov 4, 2025 •

edited

Loading

DannyLiCom commented Nov 4, 2025 •

edited

Loading

DannyLiCom commented Nov 6, 2025 •

edited

Loading